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VEHICLE BASED DATA COLLECTION AND PROCESSING SYSTEM 

CROSS-REFERENCE TO RELATED APPLICATION 

5 [0001] Applicants claim priority to United States Provisional Patent Application Serial No. 
60/412,504 filed September 20, 2002 for "Vehicle Based Data Collection and Processing 
System." 

TECHNICAL FIELD OF THE INVENTION 

10 (0002] The present invention relates, generally, to the field of remote imaging techniques and, 
more particularly, to a system for rendering high-resolution, high accuracy, low distortion digital 
images over very large fields of view. 

BACKGROUND OF THE INVENTION 

[0003] Remote sensing and imaging are broad-based technologies having a number of diverse 

15 and extremely important practical applications - such as geological mapping and analysis, 

military surveillance and planning, and meteorological forecasting. Aerial and satellite-based 

photography and imaging are especially useful remote imaging techniques that have, over recent 

years, become heavily reliant on the collection and processing of digital image, spectral, spatial, 

elevation, and vehicle location and orientation data. Spatial data - characterizing real estate 

20 improvements and locations, roads and highways, environmental hazards and conditions, utilities 

infi:astructures (e.g., phone lines, pipelines), and geophysical features - can now be collected, 
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processed, and communicated in a digital format to conveniently provide highly accurate 
mapping and surveillance data for various applications (e.g., dynamic GPS mapping). Elevation 
data may be used to improve the overall system's spatial and positional accuracy and may be 
acquired from either existing Digital Elevation Model (DEM) data sets or collected with the 
5 spectral sensor data from an active, radiation measuring Doppler based devices, or passive, 
stereographic calculations. 

[0004] Major challenges facing some such remote sensing and imaging applications are spatial 
resolution and spectral fidelity. Photographic issues, such as spherical aberrations, astigmatism, 
field curvature, coma, distortion, and chromatic aberrations are well known problems that must 

10 be dealt with in any sensor/imaging application. Certain applications require very high image 
resolution - often with tolerances of inches. Depending upon the particular system used (e.g., 
aircraft, satellite, or space vehicle), an actual digital imaging device may be located anywhere 
from several feet to miles from its target, resulting in a very large scale factor. Providing images 
with very large scale factors, that also have resolution tolerances of inches, poses a challenge to 

15 even the most robust imaging system. Thus, conventional systems usually must make some 
trade-off between resolution quality and the size of a target area that can be imaged. If the 
system is designed to provide high-resolution digital images, then the field of view (FOV) of the 
imaging device is typically small. If the system provides a larger FOV, then usually the 
resolution of the spectral and spatial data is decreased and distortions are increased. 

20 [0005] Ortho-imaging is an approach that has been used in an attempt to address this problem. 

In general, ortho-imaging imaging renders a composite image of a target by compiling varying 
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sub-images of the target. Typically, in aerial imaging applications, a digital imaging device that 
has a finite range and resolution records images of fixed subsections of a target area sequentially. 
Those images are then aligned according to some sequence to render a composite of a target 
area. 

5 [0006] Often, such rendering processes are very time-consuming and labor intensive. In many 
cases, those processes require iterative processing that measurably degrades image quality and 
resolution - especially in cases where thousands of sub-images are being rendered. In cases 
where the imaging data can be processed automatically, that data is often repetitively 
transformed and sampled - reducing color fidelity and image sharpness with each successive 

10 manipulation. If automated correction or balancing systems are employed, such systems may be 
susceptible to image anomalies (e.g., unusually bright or dark objects) - leading to over or under- 
corrections and unreliable interpretations of image data. In cases where manual rendering of 
images is required or desired, time and labor costs are immense. 

[0007] There is, therefore, a need for an ortho-image rendering system that provides efficient 
15 and versatile imaging for very large FOVs and associated data sets, while maintaining image 
quality, accuracy, positional accuracy and clarity. Additionally, automation algorithms are 
applied extensively in every phase of the planning, collecting, navigating, and processing all 
related operations. 
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SUMMARY OF THE INVENTION 
[0008] The present invention relates to a vehicle data collection and processing system using 
remote sensors. The system may include computer console units that control vehicle and system 
operations in real-time. The system may also include global positioning systems that are linked 
5 to and communicate with the computer consoles. Additionally, cameras and/or camera array 
assembUes can be employed for producing an image of a target viewed through an aperture. The 
camera array assemblies are communicatively connected to the computer consoles. The camera 
array assembly has a mount housing, a first imaging sensor centrally coupled to the housing 
having a first focal axis passing through the aperture. The camera array assembly also has a 

10 second imaging sensor coupled to the housing and offset from the first imaging sensor along an 
axis, that has a second focal axis passing through the aperture and intersecting the first focal axis 
within an intersection area. The camera array assembly has a third imaging sensor, coupled to 
the housing and offset from the first imaging sensor along the axis, opposite the second imaging 
sensor, that has a third focal axis passing through the aperture and intersecting the first focal axis 

15 within the intersection area. Any number of one-to-n cameras may be used in this maimer, 
where n can be any odd or even number. 

[0009] The system may also include an Attitude Measurement Unit (AMU) such as inertial, 
optical, or similar measurement units communicatively connected to the computer consoles and 
the camera array assemblies. The AMU may determine the yaw, pitch, and/or roll of the aircraft 
20 at any instant in time and successive DGPS positions may be used to measure the vehicle 
heading with relation to geodesic north. The AMU data is integrated with the precision DGPS 
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data to produce a robust, real-time AMU system. The system may further include a mosaicing 
module housed within the computer consoles. The mosaicing module includes a first component 
for performing initial processing on an input image. The mosaicing module also includes a 
second component for determining geographical boundaries of an input image with the second 

5 component being cooperatively engaged with the first component. The mosaicing module 
further includes a third component for mapping an input image into the composite image with 
accurate geographical position. The third component being cooperatively engaged with the first 
and second components. A fourth component is also included in the mosaicing module for 
balancing color of the input images mapped into the composite image. The fourth component 

10 can be cooperatively engaged with the first, second and third components. Additionally, the 
mosaicing module can include a fifth component for blending borders between adjacent input 
images mapped into the composite image. The fifth component being cooperatively engaged 
with the first, second, third and fourth components. 

[0010] A sixth component, an optional forward oblique and/or optional rear oblique camera 
15 array system may be implemented that collects oblique image data and merges the image data 
with attitude and positional measurements in order to create a digital elevation model using 
stereographic techniques. Creation of which may be performed in real-time onboard the vehicle 
or post processed later. This sixth component works cooperatively with the other components. 
All components may be mounted to a rigid platform for the purpose of providing co-registration 
20 of sensor data. Vibrations, turbulence, and other forces may act on the vehicle in such a way as 
to create errors in the aligrmient relationship between sensors. Utihzation of common, rigid 
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platfomi mount for the sensors provides a significant advantage over other systems that do not 
use this co-registration architecture. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0011] For a better xmderstanding of the invention, and to show by way of example how the 
5 same may be carried into effect, reference is now made to the detailed description of the 
invention along with the accompanying figures in which corresponding numerals in the different 
figures refer to corresponding parts and in which: 

FIGURE 1 illustrates a vehicle based data collection and processing system of the present 
invention; 

10 FIGURE 2 illustrates a vehicle based data collection and processing system of FIGURE 1 

with the camera array assembly of the present invention shown in more detail; 

FIGURE 3 illustrates a camera array assembly in accordance with certain aspects of the 
present invention; 

FIGURE 4 illustrates one embodiment of an imaging pattem retrieved by the camera 
1 5 array assembly of Figure 1 ; 

FIGURE 5 depicts an imaging pattem illustrating certain aspects of the present invention; 
FIGURE 6 illustrates an image strip in accordance with the present invention; 
FIGURE 7 illustrates another embodiment of an image strip in accordance with the 
present invention; 
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FIGURE 8 illustrates one embodiment of an imaging process in accordance with the 
present invention; 

FIGURE 9 illustrates diagranmiatically how photos taken with the camera array assembly 
can be aligned to make an individual frame; 
5 FIGURE 10 is a block diagram of the processing logic according to certain embodiments 

of the present invention; 

FIGURE 11 is an illustration of lateral oversampling looking down from a vehicle 
according to certain embodiments of the present invention; 

FIGURE 12 is an illustration of lateral oversampling looking down from a vehicle 
10 according to certain embodiments of the present invention; 

FIGURE 13 is an illustration of flight line oversampling looking down from a vehicle 
according to certain embodiments of the present invention; 

FIGURE 14 is an illustration of flight line oversampling looking down from a vehicle 
according to certain embodiments of the present invention; 
15 FIGURE 15 is an illustration of progressive magnification looking down from a vehicle 

according to certain embodiments of the present invention; 

FIGURE 16 is an illustration of progressive magnification looking down from a vehicle 
according to certain embodiments of the present invention; 

FIGURE 17 is an illustration of progressive magnification looking down from a vehicle 
20 according to certain embodiments of the present invention; and 
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FIGURE 18 is a schematic of the system architecture according to certain embodiments 
of the present invention. 

DETAILED DESCRIPTION OF THE INVENTION 
5 [0012] While the making and using of various embodiments of the present invention are 
discussed in detail below, it should be appreciated that the present invention provides many 
apphcable inventive concepts, which can be embodied in a wide variety of specific contexts. 
The specific embodiments discussed herein are merely illustrative of specific ways to make and 
use the invention and do not limit the scope of the invention. 

10 [0013] A vehicle based data collection and processing system 100 of the present invention is 
shown in Figure 1. Additional aspects and embodiments of the present invention are shown in 
Figures 2 and 18. System 100 includes one or more computer consoles 102. The computer 
consoles contain one or more computers 104 for controlling both vehicle and system operations. 
Examples of the fimctions of the computer console are the controlling digital color sensor 

15 systems that can be associated with the data collection and processing system, providing the 
display data to a pilot, coordinating the sateUite generated GPS pulse-per-second (PPS) event 
trigger (which may be 20 or more pulses per second), data logging, sensor control and 
adjustment, checking and alarming for error events, recording and indexing photos, storing and 
processing data, flight planning capability that automates the navigation of the vehicle, data, and 

20 providing a real-time display of pertinent information. A communications interface between the 
control computer console and the vehicle autopilot control provides the ability to actually control 
the flight path of the vehicle in real-time. This results in a more precise control of the vehicle's 
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path than is possible by a human being. All of these functions can be accomphshed by the use of 
various computer programs that are synchronized to the GPS PPS signals and take into account 
the various electrical latencies of the measurement devices. 

[0014] One or more differential global positioning systems 106 are incorporated into the 
5 system 100. The global positioning systems 106 are used to navigate and determine precise 
flights paths during vehicle and system operations. To accomplish this, the global positioning 
systems 106 are communicatively linked to the computer console 102 such that the information 
from the global positioning systems 106 can be acquired and processed without flight 
interruption. Zero or more GPS \mits may be located at known survey points in order to provide 
10 a record of each sub-seconds' GPS satellite-based errors in order to be able to back correct the 
accuracy of the system 100. GPS and/or ground based positioning services may be used that 
eliminate the need for ground control points altogether. This technique results in greatly 
improved, sub-second by sub-second positional accuracy of the data capture vehicle. 

[0015] One or more AMUs 108 that provide real-time yaw, pitch, and roll information that is 

15 used to accurately determine the attitude of the vehicle at the instant of data capture are also 

communicatively linked to the computer console 102. The present attitude measurement unit 

(AMU) (e.g. Applanix POS AV), uses three high performance fiber optic gyros, one gyro each 

for yaw, pitch, and roll measurement. AMUs from other manufacturers, and AMUs that use 

other inertial measurement devices can be used as well. Additionally, an AMU may be 

20 employed to determine the instantaneous attitude of the vehicle and make the system more fault 

tolerant to statistical errors in AMU readings. Connected to the AMU can be one or more multi- 
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frequency DGPS receivers 110. The multi-frequency DGPS receivers 110 can be integrated with 
the AMU's yaw, pitch, and roll attitude data in order to more accurately determine the location 
of the remote sensor platform in three dimensional space. Additionally, the direction of geodesic 
North may be determined by the vector created by successive DGPS positions, recorded in a 
5 synchronized manner with the GPS PPS signals. 

[0016] One or more one or more camera array assemblies 112 for producing an image of a 
target viewed through an aperture are also communicatively connected to the one or more 
computer consoles 102. The camera array assembUes 112, which will be described in greater 
detail below, provide the data collection and processing system with the ability to capture high 
10 resolution, high precision progressive scan or line scan, color digital photography, 

[0017] The system may also include DC power and conditioning equipment 114 to condition 
DC power and to invert DC power to AC power in order to provide electrical power for the 
system. The system may ftirther include a navigational display 116, which graphically renders 
the position of the vehicle versus the flight plan for use by the pilot (either onboard or remote) of 

15 the vehicle to enable precision flight paths in horizontal and vertical planes. The system may 
also include an EMU module comprised of LIDAR, SAR 118 or a forward and rear obUque 
camera array for capturing three dimensional elevation/relief data. The EMU module 118 can 
include a laser unit 120, an EMU control unit 122, and an EMU control computer 124. 
Temperature controlling devices, such as soUd state cooling modules, can also be deployed as 

20 needed in order to provide the proper thermal environment for the system. 

11 
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[0018] The system also includes a mosaicing module, not depicted, housed with the computer 
console 102. The mosaicing module, which will be described in further detail below, provides 
the system the ability to gather data acquired by the global positioning system 106, the AMU 
108, and the camera system 1 12 and process that data into useable orthomaps, 

5 [0019] The system 100 also can include a Self-Locking flight pattern technique that provides 
the ability to micro-correct the positional accuracy of adjacent flight paths in order to realize 
precision that exceeds the native precision of the AMU and DGPS sensors alone. 

[0020] A complete flight planning methodology is used to micro plan all aspects of missions. 
The inputs are the various mission parameters (latitude/longitude, resolution, color, accuracy, 

10 etc.) and the outputs are detailed on-line digital maps and data files that are stored onboard the 
data collection vehicle and used for real-time navigation and alarms. The ability to interface the 
flight planning data directly into the autopilot is an additional integrated capability. A computer 
program may be used that automatically controls the flight path, attitude adjustments, graphical 
display, moving maps of the vehicle path, checks for alarm conditions and corrective actions, 

15 notifies the pilot and/or crew of overall system status, and provides for fail-safe operations and 
controls. Safe operations parameters may be constantly monitored and reported. Whereas the 
current system uses a manned crew, the system is designed to perform equally will in an 
unmanned vehicle. 

[0021] Figure 2 shows another depiction of the present invention. In Figure 2, the camera 
20 array assembly 1 12 is shown in more detail. As is shown, the camera array assembly 112 allows 
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for images to be acquired from the rear oblique, the forward obliques and the nadir positions. 
Figure 3, describes in more detail a camera array assembly of the present invention. Figure 3 
provides a camera array assembly 300 airborne over target 302 (e.g., terrain). For illustrative 
purposes, the relative size of assembly 300, and the relative distance between it and terrain 302, 
5 are not depicted to scale in Figure 3. The camera array assembly 300 comprises a housing 304 
within which imaging sensors 306, 308, 310, 312 and 314 are disposed along a concave 
curvilinear axis 316. The radius of curvature of axis 316 may vary or be altered dramatically, 
providing the ability to effect very subtle or very drastic degrees of concavity in axis 316. 
Alternatively, axis 316 may be completely linear - having no curvature at all. The imaging 
10 sensors 306, 308, 310, 312 and 314 couple to the housing 304, either directly or indirectly, by 
attachment members 318. Attachment members 318 may comprise a number of fixed or 
dynamic, permanent or temporary, connective apparatus. For example, the attachment members 
318 may comprise simple welds, removable clamping devices, or electro-mechanically 
controlled universal joints. 

15 [0022] Additionally, the system 100 may have a real-time, onboard navigation system to 
provide a visual, bio-feedback display to the vehicle pilot, or remote display in the case of 
operations in an immanned vehicle. The pilot is able to adjust the position of the vehicle in real- 
time in order to provide a more accurate flight path. The pilot may be onboard the vehicle or 
remotely located and using the flight display to control the vehicle through a communication 

20 link. 
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[0023] The system 100 may also use highly fault-tolerant methods been developed that 
provides a software inter-leaved disk storage methodology that allows one or two hard drives to 
fail and still not lose target data that is stored on the drives. This software inter-leaved disk 
storage methodology provides superior fault-tolerance and portability versus other, hardware 
5 methodologies, such as RAID-5. 

[0024] The system 100 may also incorporate a methodology that has been developed that 
allows for a short calibration step just before mission data capture. The calibration methodology 
step adjusts the camera settings, mainly exposure time, based on sampling the ambient light 
intensity and setting near optimal values just before reaching the region of interest. A moving 

10 average algorithm is then used to make second-by-second camera adjustments in order to deliver 
improved, consistent photo results. This improves the color processing of the orthomaps. 
Additionally, the calibration may be used to check or to establish the exact spatial position of 
each sensor device (cameras, DPG, AMU, EMU, etc.). In this manner, changes that may happen 
in the spatial location of these devices may be accounted for and maintain overall system 

1 5 precision metrics. 

[0025] Additionally, the system 100 may incorporate a methodology that has been developed 
that allows for caUbrating the precision position and attitude of each sensor device (cameras, 
DPG, AMU, EMU, etc.) on the vehicle by flying over an area that contains multiple known, 
visible, highly accurate geographic positions. A program takes this data as input and outputs the 
20 micro positional data that is then used to precisely process the orthomaps. 

14 
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[0026] As depicted in Figure 3, housing 304 comprises a simple enclosure inside of which 
imaging sensors 306, 308, 310, 312 and 314 are disposed. Whereas Figure 3 depicts a 5-camera 
array, the system works equally well when utiUzing any number of camera sensors from 1 to any 
number. Sensors 306 - 314 couple, via the attachment members 318, either collectively to a 
5 single transverse cross member, or individually to lateral cross members disposed between 
opposing walls of the housing 304. In ahemative embodiments, the housing 304 may itself 
comprise only a supporting cross member of concave curvature to which the imaging sensors 
306 through 314 couple, via members 318. In other embodiments, the housing 304 may 
comprise a hybrid combination of enclosure and supporting cross member. The housing 304 

10 further comprises an aperture 320 formed in its surface, between the imaging sensors and target 
302. Depending upon the specific type of host craft, the aperture 320 may comprise only a void, 
or it may comprise a protective screen or window to maintain environmental integrity within the 
housing 304. In the event that a protective transparent plate is used for any sensor, special 
coatings may be apphed to the plate to improve the quality of the sensor data. Optionally, the 

15 apertxu-e 320 may comprise a lens or other optical device to enhance or alter the nature of the 
images recorded by the sensors. The aperture 320 is formed with a size and shape sufficient to 
provide the imaging sensors 306 through 314 proper lines of sight to a target region 322 on 
terrain 302. 

[0027] The imaging sensors 306 through 314 are disposed within or along housing 304 such 
20 that the focal axes of all sensors converge and intersect each other within an intersection area 
bounded by the aperture 320. Depending upon the type of image data being collected, the 

15 
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specific imaging sensors used, and other optics or equipment employed, it may be necessary or 
desirable to offset the intersection area or point of convergence above or below the aperture 320. 
The imaging sensors 306 through 314 are separated from each other at angular intervals. The 
exact angle of displacement between the imaging sensors may vary widely depending upon the 
5 number of imaging sensors utiUzed and on the type of imaging data being collected. The angular 
displacement between the imaging sensors may also be unequal, if required, so as to provide a 
desired image offset or alignment. Depending upon the number of imaging sensors utilized, and 
the particular configuration of the array, the focal axes of all imaging sensors may intersect at 
exactly the same point, or may intersect at a plurality of points, all within close proximity to each 
10 other and within the intersection area defined by the aperture 320. 

[0028] As depicted in Figure 3, the imaging sensor 310 is centrally disposed within the 
housing 304 along axis 316. The imaging sensor 310 has a focal axis 324, directed orthogonally 
fi-om the housing 304 to align the line of sight of the imaging sensor with the image area 326 of 
the region 322. The imaging sensor 308 is disposed within the housing 304 along the axis 316, 

15 adjacent to the imaging sensor 310. The imaging sensor 308 is aligned such that its line of sight 
coincides with the image area 328 of the region 322, and such that its focal axis 330 converges 
with and intersects the axis 324 within the area bounded by the aperture 320. The imaging 
sensor 312 is disposed within the housing 304 adjacent to the imaging sensor 310, on the 
opposite side of the axis 316 as the imaging sensor 308. The imaging sensor 312 is aligned such 

20 that its line of sight coincides with the image area 332 of the region 322, and such that its focal 
axis 334 converges with and intersects axes 324 and 330 within the area bounded by the aperture 

16 
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320. The imaging sensor 306 is disposed within the housing 304 along the axis 316, adjacent to 
the sensor 308. The imaging sensor 306 is aligned such that its line of sight coincides with the 
image area 336 of region 322, and such that its focal axis 338 converges with and intersects the 
other focal axes within the area bounded by aperture 320. The imaging sensor 314 is disposed 
5 within housing 304 adjacent to sensor 312, on the opposite side of axis 316 as sensor 306. The 
imaging sensor 314 is ahgned such that its line of sight coincides with image area 340 of region 
322, and such that its focal axis 344 converges with and intersects the other focal axes within the 
area bounded by aperture 320. 

[0029] The imaging sensors 306 through 314 may comprise a number of digital imaging 
10 devices including, for example, individual area scan cameras, line scan cameras, infrared 
sensors, hyperspectral and/or seismic sensors. Each sensor may comprise an individual imaging 
device, or may itself comprise an imaging array. The imaging sensors 306 through 314 may all 
be of a homogenous nature, or may comprise a combination of varied imaging devices. For ease 
of reference, the imaging sensors 306 through 314 are hereafter referred to as cameras 306 
1 5 through 314, respectively. 

[0030] In large- format fihn or digital cameras, lens distortion is typically a source of imaging 
problems. Each individual lens must be carefully calibrated to determine precise distortion 
factors. In one embodiment of this invention, small-format digital cameras having lens angle 
widths of 17 degrees or smaller are utiUzed. This alleviates noticeable distortion efficiently and 
20 affordably. 
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[0031] Cameras 306 through 314 are alternately disposed within housing 304 along axis 316 
such that each camera's focal axis converges upon aperture 320, crosses focal axis 324, and 
aUgns its field of view with a target area opposite its respective position in the array resulting in a 
"cross-eyed", retinal relationship between the cameras and the imaging target(s). The camera 
5 array assembly 300 is configured such that adjoining borders of image areas 326, 328, 332, 336 
and 340 overlap slightly. 

[0032] If the attachment members 318 are of a permanent and fixed nature (e.g., welds), then 
the spatial relationship between aperture 320, the cameras, and their lines of sight remain fixed 
as will the spatial relationship between image areas 326, 328, 332, 336 and 340. Such a 

10 configuration may be desirable in, for example, a satellite surveillance application where the 
camera array assembly 300 will remain at an essentially fixed distance fi-om region 322. The 
position and alignment of the cameras is set such that areas 326, 328, 332, 336 and 340 provide 
fiiU imaging coverage of region 322. If the attachment members 318 are of a temporary or 
adjustable nature, however, it may be desirable to selectively adjust, either manually or by 

15 remote automation, the position or alignment of the cameras so as to shift, narrow or widen areas 
326, 328, 332, 336 and 340 - thereby enhancing or altering the quality of images collected by the 
camera array assembly 300. 

[0033] Camera 310 is designated as the principal camera. The image plane 326 of camera 310 
serves as a plane of reference. The orientations of the other cameras 306, 308, 312 and 314 are 
20 measured relative to the plane of reference. The relative orientations of each camera are 
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measured in terms of the yaw, pitch and roll angles required to rotate the image plane of the 
camera to become parallel to the plane of reference. The order of rotations is roll, pitch and yaw. 

[0034] Referring now to Figure 2, images of areas 336, 328, 326, 332 and 340 taken by 
cameras 306 through 314, respectively, are illustrated from an overhead view. Again, because of 
5 the "cross-eyed" arrangement, the image of area 336 is taken by camera 306, the image of area 
340 is taken by camera 314, and so on. In one embodiment of the present invention, images 
other than those taken by the center camera 310 take on a trapezoidal shape after perspective 
transformation. Cameras 306 through 314 form an array along axis 316 that is, in most 
appHcations, pointed down vertically. In an alternative embodiment, a second array of cameras, 

10 configured similar the array of cameras 306 through 314, is aligned with respect to the first array 
of cameras to have an oblique view providing a "heads-up" perspective. The angle of 
declination from horizontal of the heads-up camera array assembly may vary due to mission 
objectives and parameters but angles of 25-45 degrees are typical. Other altemative 
embodiments, varying the mounting of camera arrays, are similarly comprehended by the present 

15 invention. In all such embodiments, the relative positions and attitudes of the cameras are 
precisely measured and calibrated so as to facilitate image processing in accordance with the 
present invention. 

[0035] In one embodiment of the present invention, an extemal mechanism (e.g., a GPS 

timing signal) is used to trigger the cameras simultaneously thereby capturing an array of input 

20 images. A mosaicing module then renders the individual input images from such an array into 

an ortho-rectified compound image (or "mosaic"), without any visible seams between the 

19 
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adjacent images. The mosaicing module perfonns a set of tasks comprising: detemiining the 
geographical boundaries and dimensions of each input image; projecting each input image onto 
the mosaic with accurate geographical positioning; balancing the color of the images in the 
mosaic; and blending adjacent input images at their shared seams. The exact order of the tasks 
5 performed may vary, depending upon the size and nature of the input image data. In certain 
embodiments, the mosaicing module performs only a single transformation to an original input 
image during mosaicing. That transformation can be represented by a 4 x 4 matrix. By 
combining multiple transformation matrices into a single matrix, processing time is reduced and 
original input image sharpness is retained. 

10 [00361 During mapping of the input images to the mosaic, especially when mosaicing is 
performed at high resolutions, pixels in the mosaic (i.e., output pixels) may not be mapped to by 
any pixels in the input images (i.e., input pixels). Warped lines could potentially result as 
artifacts in the mosaic. Certain embodiments of the present invention overcome this with a 
super-sampling system, where each input and output pixel is further divided into mnxm grid of 

15 sub-pixels. Transformation is performed from sub-pixels to sub-pixels. The final value of an 
output pixel is the average value of its sub-pixels for which there is a corresponding input sub- 
pixel. Larger n and m values produce mosaics of higher resolution, but do require extra 
processing time. 

[0037] During its processing of image data, the mosaicing module may utilize the following 

20 information: the spatial position (e.g., x, y, z coordinates) of each camera's focal point at the 

time an input image is captured; the attitude (i.e., yaw, pitch, roll) of each camera's image plane 

20 
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relative to the target region's ground plane at the time an input image was captured; each 
camera's fields of view (i.e., along track and cross track); and the Digital Terrain Model (DTM) 
of the area. The attitude can be provided by the AMUs associated with the system. Digital 
terrain models (DTMs) or Digital surface models (DSMs) can be created from information 
5 obtained using a LIDAR module 118. LIDAR is similar to the more familiar radar, and can be 
thought of as laser radar. In radar, radio waves are transmitted into the atmosphere that scatters 
some of the energy back to the radar's receiver. LIDAR also transmits and receives 
electromagnetic radiation, but at a higher frequency since it operates in the ultraviolet, visible 
and infrared region of the electromagnetic spectrum. In operation, LIDAR transmits light out to 
10 a target area. The transmitted Ught interacts with and is changed by the target area. Some of this 
light is reflected / scattered back to the LIDAR instrument where it can be analyzed. The change 
in the properties of the Ught enables some property of the target area to be determined. The time 
for the light to travel out to the target area and back to LIDAR device is used to determine the 
range to the target. 

15 [0038] DTM and DSM data sets can also be captured from the camera array assembly. 
Traditional means of obtaining elevation data may also be used such as stereographic techniques. 

[0039] There are presently three basic types of LIDAR: Range finders, Differential Absorption 

LIDAR (DIAL) and Doppler LIDAR. Range finder LIDAR is the simplest LIDAR and is used 

to measure the distance from the LIDAR device to a soUd or hard target. DIAL LIDAR is used 

20 to measure chemical concentrations (such as ozone, water vapor, pollutants) in the atmosphere. 

A DIAL LIDAR uses two different laser wavelengths that are selected so that one of the 

21 



Attorney Docket No. 127976-1000 PATENT 

wavelengths is absorbed by the molecule of interest while the other wavelength is not. The 
difference in intensity of the two return signals can be used to deduce the concentration of the 
molecule being investigated. Doppler LIDAR is used to measure the velocity of a target. When 
the hght transmitted from the LIDAR hits a target moving towards or away from the LIDAR, the 
5 wavelength of the light reflected/scattered off the target will be changed sUghtly. This is known 
as a Doppler-shift and therefore Doppler LIDAR. If the target is moving away from the LIDAR, 
the retum light will have a longer wavelength (sometimes referred to as a red shift), if moving 
towards the LIDAR the retum Kght will be at a shorter wavelength (blue shifted). The target can 
be either a hard target or an atmospheric target (e.g. microscopic dust and aerosol particles that 
10 are carried by the wind. 

[0040] A camera's focal point is used as a perspective transformation center. Its position in 
space is determined, for example, by a multi-frequency carrier phase post-processed GPS system 
mounted on the host craft. The offsets, in three dimensions, of a camera's focal point must be 
carefiiUy measured against the center of the GPS antenna. These offsets are combined with the 
15 position of the GPS antenna, and the orientation of the host craft, to determine the exact position 
of the camera's focal point. The position of the GPS antenna is determined by post-flight 
processing of collected GPS data against similar ground-based GPS antennas deployed at 
precisely surveyed points. 

[0041] One or more AMUs (e.g., the Applanix POS AV) are mounted onboard for attitude 

20 determination. The attitude of the AMU reference plane relative to the target region's ground 

plane is measured and recorded at short intervals, with accuracy better than one-hundredth of one 
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degree. The attitude of the AMU reference plane is defined as the series of rotations that can be 
performed on the axes of this plane to make it parallel to the ground plane. The term "align" is 
also used to describe this operation. 

[0042] The attittide of center camera 310 (i.e. its unage plane), relative to the AMU, must be 
5 carefully caUbrated. The attittide of each of the other cameras, relative to center camera 310, 
must also be carefully calibrated. This dependent calibration is more efficient than directly 
calibrating each camera. When the camera array assembly 300 is remounted, only center camera 
310 needs to be recalibrated. Effectively, a series of two transformations is appUed to an input 
image fi-om center camera 310. First, the center camera's image plane is aligned to the AMU 
10 plane. Then, the AMU plane is aligned again to the ground plane. These tt-ansformations, 
however, combine into a single operation by multiplying their respective transformation 
mattices. For images from each of the other cameras, an additional transformation is first 
performed to align it with the center camera's image plane. 

[00431 The position of the focal point of center camera 310 is determined as described above. 

15 The X and y components of this position determine the position of the mosaic's nadir point 400 
on the ground. Field of view (FOV) angles of each camera are known, thus the dimensions of 
each input image can be determined by the z component of that camera's focal point. An 
average elevation of the ground is determined by computing the average elevation of points in 
the DTMs of the area, and then each input image is projected to an imaginary horizontal plane at 

20 this elevation. Relief displacement is then applied using the DTMs of the area. The DTMs can 

be obtained fi-om many sources including: the USGS 30- or 10-meter DTMs available for most 
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of the US; commercial DTMs; or DTMs obtained by a LIDAR or SAR EMU device mounted on 
the host craft that captures data concurrently with the cameras. 

[0044] Besides being geographically correctly placed, the resulting compound image also 
needs to have color consistency throughout, and no visible seams at the joints between two 
5 adjacent images. The present invention provides a number of techniques for achieving this goal. 

[0045] A characteristic of a conventional camera is the exposure time (i.e., the time the shutter 
is open to collect light onto the image plane). The longer the exposure time, the lighter the 
resultant image becomes. Exposure time must adapt to changes in ambient Ughting caused by 
conditions such as: cloud coverage; the angle and position of the sun relative to the camera; and 

10 so forth. Optimal exposure time may also depend on a camera's orientation with respect to 
lighting sources (e.g., cameras pointing towards a sunht object typically receive more ambient 
light than those pointing towards a shaded object). Exposure time is adjusted to keep the average 
intensity of an image within a certain desired range. For example, in 24-bit color images each 
Red, Green and Blue component can have intensity values from 0 to 255. In most instances, 

15 however, it is desirable to keep the average intensity at a mean value (i.e., 127). 

[0046] In the present invention, an exposure control module controls exposure time for each of 
the cameras or imaging sensors. It examines each input image and calculates average image 
intensity. Based on a moving average (i.e., average intensity of the last ^number of images), the 
exposure control module determines whether to increase or decrease exposure time. The module 
20 can use a longer running average to effect a slower reaction to changes in lighting conditions, 
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with less susceptibility to unusually dark or light images (e.g., asphalt roads or water). The 
exposure control module controls exposure time for each camera separately. 

[0047] In systems where cameras are moxmted without forward-motion compensation 
mechanisms, there must be a maximum limit for exposure time. Setting exposure time to a value 
5 larger than the maximum may cause motion-induced blurriness. For example, assume cameras 
are mounted on an airplane traveling at 170 miles/hour (or about 3 inches/ms). Assume desired 
pixel resolution is 6 inches. Forward motion during image capture should be limited to half a 
pixel size - which in this case equals 3 inches. Thus, maximum exposure for example is 1 
millisecond. 

10 [0048] In controlling imaging quality, it is useful to be able to determine if changes in light 
intensity are caused either due to a change in ambient light or due to the presence of unusually 
light or dark objects (e.g., reflecting water body, metal roofs, asphalts, etc.). Certain applications 
of this invention involve aerial photography or surveillance. It is observed that aerial images of 
the ground usually contain plants and vegetation - which have more consistent reflectivity than 

15 water bodies or man-made structures such as roads and buildings. Of course, images of plants 
and vegetation are usually green-dominant (i.e., the green component is the greatest of the red, 
green and blue values). Therefore, intensity correlation can be made more accurate by focusing 
on the green-dominant pixels. 

[0049] The exposure control module computes the average intensity of an image by selecting 
20 only green-dominant pixels. For example, if an image has 1 million pixels and 300,000 are 
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green-dominant, only those 300,000 green-dominant pixels are included in the calculation of 
average intensity. This results in an imaging process that is less susceptible to biasing caused by 
man-made structures and water bodies, whose pixels are usually not green-dominant. As 
previously noted, it is desirable to maintain an intensity value of about 127. When intensity 
5 value is over 127 (i.e., over-exposed), exposure time is reduced so that less light is captured. 
Similarly, when intensity value is under 127 (i.e., under-exposed), exposxu*e time is increased so 
that more light is captured. For example, consider a system flying over a target terrain area 
having many white roofs, whose intensities are very high. Average intensity for the images 
captured would tend to be high. In most conventional systems, exposure time would by reduced 

10 in order to compensate. In such an example, however, reducing exposure time is not proper, 
because the average intensity of the images has been biased by the bright roofs. Reducing 
exposure time would result in images where the ground is darker than it should be. In contrast, if 
only green-dominant pixels are processed in accordance with the present invention, then pixels 
representing the overly bright roofs do bias the average intensity and the exposure time is not 

15 changed. 

[0050] Thus, the exposure control module reduces intensity differences between input images. 
Nonetheless, further processing is provided to enhance tonal balance. There are a number of 
factors (e.g., lens physics, atmospheric conditions, spatial/positional relationships of imaging 
devices) that cause an xmeven reception of light from the image plane. More light is received in 
20 the center of a camera or sensor than at the edges. 
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[0051] The mosaicing module of the present invention addresses this with an anti-vignetting 
function, illustrated in reference now to Figure 5. A number of focal columns 500, 502, 504, 506 
and 508 converge from image plane 509 and cross through focal point 510 as they range across 
imaging target area 512 (e.g., ground terrain). Columns 500 through 508 may comprise 
individual resolution columns of a single camera or sensor, or may represent the focal axes of a 
nimiber of independent cameras or sensors. For reference purposes, column 504 serves as the 
axis and point 513 at which column 504 intersects image plane 509 serves as a principal point. 
The exposure control module applies an anti-vignetting function multiplying the original 
intensity of an input pixel with a column-dependent anti-vignetting factor. Because the receiving 
surface is represented as a plane with a coordinate system, each column will have a number of 
resolution rows (not shown). This relationship may be expressed, for a pixel p at column x and 
rowj^, as follows: 

<adjusted intensity> = <original intensity> *y(x); 

wherey(x) is a function of the form: 

y(x) = cos(off-axis angle) ** 4. 

The off-axis angle 514 is: zero for center column 504; larger for columns 502 and 506; and 
larger still for columns 500 and 508. The overall field of view angle 516 (FOVx angle) is 
depicted between columns 504 and 508. 
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[0052] The fiinctionXx) can be approximated by a number of line segments between colunms. 
For a point falling within a line segment between any given columns cl and c2, an adjustment 
factor is computed as follows: 

<adjustment factor for c> =y(cl) + [/(c2) -/cl) * (c - cl)/(c2-cl)]; 

5 [0053] where y(cl) andy(c2) are the / function values of the off-axis angles at column cl and 
c2, respectively. 

[0054] Each set of input images needs to be stitched into a mosaic image. Even though the 
exposure control module regulates the amount of Ught each camera or sensor receives, the 
resulting input images may still differ in intensity. The present invention provides an intensity- 

10 balancing module that compares overlapping area between adjacent input images, to further 
balance the relative intensities. Because adjoining input images are taken simultaneously, the 
overlapping areas should, in theory, have identical intensity in both input images. However, due 
to various factors, the intensity values are usually not the same. Some such factors causing 
intensity difference could include, for example, the exposure control module being biased by 

15 unusually bright or dark objects present in the field of view of only a particular camera, or the 
boresight angles of cameras being different (i.e., cameras that are more slanted receive less light 
than those more vertical). 

[0055] To balance two adjacent images, one is chosen as the reference image and the other is 

the secondary image. A correlation vector (fR, fG, FB) is determined using, for example, the 

20 following process. Let V be a 3 x 1 vector representing the values (R, G and B) of a pixel: 
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R 

V= G 
B . 

5 A correlation matrix C may be derived as: 

FR 0 0 
C= 0 FG 0 

0 0 FB ; 

10 

where FR = Avglr/Avgin; Avglr = Red average intensity of overlapped region in reference 
image; Avgin = Red average intensity of overlapped region in new image; and FG and FB are 
similarly derived. 

[0056] The correlation matrix scales pixel values of the secondary image so that the average 
15 intensity of the overlapping area of the secondary image becomes identical to the average 
intensity of the overlapping area of the reference image. The second image can be balanced to 
the reference image by multiplying its pixel values by the correlation matrix. 

[0057] Thus, in one embodiment of a balancing process according to the present invention, a 
center image is considered the reference image. The reference image is first copied to the 
20 compound image (or mosaic). Overlapping areas between the reference image and an adjoining 
image (e.g., the near left image) are correlated to compute a balancing correlation matrix (BCM). 
The BCM will be multipUed with vectors representing pixels of the adjoining image to make the 
intensity of the overlapping area identical in both images. One embodiment of this relationship 
may be expressed as: 

25 Let I(center) = Average intensity of overlapping area in center image; 
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I(adjoining) = Average intensity of overlap in adjoining image; then 

Balancing factor = I(center) / I(adjoimng). 

[0058] The balancing factor for each color channel (i.e., red, green and blue) is independently 
computed. These three values form the BCM. The now-balanced adjoining image is copied to 
the mosaic. Smooth transitioning at the border of the copied image is providing by "feathering" 
with a mask. This mask has the same dimension as the adjoining image and comprises a number 
of elements. Each element in the mask indicates the weight of the corresponding adjoining 
image pixel in the mosaic. The weight is zero for pixels at the boundary (i.e. the output value is 
taken from the reference image), and increases gradually in the direction of the adjoining image 
until it becomes unity - after a chosen blending width has been reached. Beyond the blending 
area, the mosaic will be entirely determined by the pixels of the adjoining image. Similarly, the 
overlaps between all the other constituent input images are analyzed and processed to compute 
the correlation vectors and to balance the intensities of the images. 

[0059] A correlation matrix is determined using, for example, the following process with 
reference to Figure 6. Figure 6 depicts a strip 600 being formed in accordance with the present 
invention. A base mosaic 602 and a new mosaic 604, added along path (or track) 606, overlap 
each other in region 608. Let V be a vector that represents the R, G and B values of a pixel: 

R 

V= G 
B 
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Let h be the transition width of region 608, and y be the along-track 606 distance from the 
boundary 610 of the overlapped region to a point A, whose pixel values are represented by V. 



Let C be the correlation matrix: 

5 



FR 


0 


0 


0 


FG 


0 


0 


0 


FB 



10 The balanced value of V, called F' is: 

K' = \y/kl + (1 -y/h ).C] x V, for 0 < j; < h; 
V' = YJory>= h; 

15 

Where I is the identity matrix 

1 0 0 
1= 0 1 0 

0 0 1. 

20 

Note that the "feathering" technique is also used in combination with the gradient to minimize 
seam visibility. 

[0060] When mosaics are long, differences in intensity at the overlap may change from one 
end of the mosaic to the other. Computing a single correlation vector to avoid creating visible 
25 seams may not be possible. The mosaic can be divided into a number of segments corresponding 
to the position of the original input images that make up the mosaic. The process described 
above is appUed to each segment separately to provide better local color consistency. 
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[00611 Under this refined algorithm, pixels at the border of two segments may create vertical 
seams (assuming north-south flight lines). To avoid this problem, balancing factors for pixels in 
this area have to be "transitioned" from that of one segment to the other. This is explained now 
with reference to Figure 7. 

5 [0062] Figure 7 depicts a strip 700 being formed in accordance with the present invention. A 
base mosaic 702 and a new segment 704 overlap in area 706. Mosaic 702 and another new 
segment 708 overlap in area 710. Segments 704 and 708 overlap in area 712, and areas 706, 710 
and 712 all overlap and coincide at area 714. For explanation purposes, point 716 serves as an 
origin for y-axis 718 and x-axis 720. Movement along y-axis 718 represents movement along 

1 0 the flight path of the imaging system. Point 7 1 6 is located at the lower left of area 7 1 4. 

[0063] According to the present invention, the dimensions of a strip are determined by the 
minimum and maximum x and 3/ values of the constituent mosaics. An output strip is initialized 
to a background color. A first mosaic is transferred to the strip. The next mosaic (along the 
flight path) is processed next. Intensity values of the overlapping areas of the new mosaic and 

15 the first mosaic are correlated, separately for each color channel. The new mosaic is divided into 
a number of segments corresponding to the original input images that made up the mosaic. A 
mask matrix, comprising a number of mask elements, is created for the new mosaic. A mask 
element contains the correlation matrix for a corresponding pixel in the new mosaic. All 
elements in the mask are initialized to unity. The size of the mask can be limited to just the 

20 transition area of the new mosaic. The correlation matrix is calculated for the center segment. 

The mask area corresponding to the center segment is processed. The values of the elements at 
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the edge of the overlap area are set to the correlation vector. Then, gradually moving away from 
the first mosaic along the strip, the components of the correlation matrix are either increased or 
decreased (whether they are less or more than unity, respectively) until they become xmity at a 
predetermined transition distance. The area of the mask corresponding to a segment adjoining 
5 the center segment is then processed similarly. However, the area 814 formed by the first mosaic 
and the center and adjoining segments of the new image requires special treatment. Because the 
correlation matrix for the adjoining segment may not be identical to that of the center segment, a 
seam may appear at the border of the two segments in the overlap area 714 with the first mosaic. 
Therefore, the comer is influenced by the correlation matrices from both segments. For a mask 
10 cell A at distance x to the border with the center segment and distance to the overlap edge, its 
correlation matrix is the distance-weighted average of the two segments, evaluated as follows: 

For pixel A(x, y) in area 714 at distance x to the border with the center segment, its 
balanced values are computed as the distance-weighted averages of the values computed 
using the two segments; 
15 VI is the balanced RGB vector based on segment 704; 

V2 is the balanced RGB vector based on segment 708; 
V is the combined (final) balanced RGB vector 

V = ((rf-x)/d).Vl + (x/rf).V2; 
Where 

20 X-axis is the line going through bottom of overlapped region; 
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y-axis is the line going through the left side of the overlapped region between 

segments 704 and 708; 

h is the transition width; and 

d is the width of the overlapped region between segments 704 and 708. 
5 The mask areas corresponding to other adjoining segments are computed similarly. 

[0064] Further according to the present invention, a color fidelity (i.e., white-balance) filter is 
applied. This multipUes R and B components with a determinable factor to enhance color 
fidehty. The factor may be determined by calibrating the cameras and lenses. The color fidehty 
filter ensures that the colors in an image retain their fidelity, as perceived directly by the human 

10 eye. Within the image capture apparatus, the Red, Green and Blue light receiving elements may 
have different sensitivities to the color they are supposed to capture. A "while-balance" process 
is applied - where image of a white object is captured. Theoretically, pixels in the image of that 
white object should have equivalent R, G and B values. In reality, however, due to different 
sensitivities and other factors, the average color values for each R, G and B may be avgR, avgG 

15 and avgB, respectively. To equalize the color components, the R, G and B values of the pixels 
are multiplied by the following ratios: 

R values are multiplied by the ratio avgG / avgR; and 

B values are multiphed by the ratio avgG / avgB. 

The end result is that the image of the white object is set to have equal RGB components. 
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[0065] In most applications, a strip usually covers a large area of non-water surface. Thus, 
average intensity for the strip is unlikely to be skewed by anomalies such as highly reflecting 
surfaces. The present invention provides an intensity normalization module that normalizes the 
average intensity of each strip so that the mean and standard deviation are of a desired value. 

5 For example, a mean of 127 is the norm in photogrammetry. A standard deviation of 51 helps to 
spread the intensity value over an optimal range for visual perception of image features. Each 
strip may have been taken in different lighting conditions and, therefore, may have different 
imaging data profiles (i.e., mean intensity and standard deviation). This module normahzes the 
strips, such that all have the same mean and standard deviation. This enables the strips to be 

1 0 stitched together without visible seams. 

[0066] This intensity normalization comprises a computation of the mean intensity for each 
channel R, G and B, and for all channels. The overall standard deviation is then computed. Each 
R, G and B value of each pixel is transformed to the new mean and standard deviation: 

new value = new mean + (old value - old mean) * (new std/old std). 

15 [0067] Next, multiple adjacent strips are combined to produce tiled mosaics for an area of 
interest. Finished tiles can correspond to the USGS quads or quarter-quads. Stitching strips into 
mosaics is similar to stitching mosaics together to generate strips, with strips now taking the role 
of the mosaics. At the seam line between two strips, problems may arise if the line crosses 
elevated structures such as buildings, bridges, etc. This classic problem in photogrammetry 

20 arises fi-om the parallax caused by the same object being looked at firom two different 
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perspectives. During imaging of a building, for example, one strip may present a view from one 
side of the building while another strip presents a view from another side of the building. After 
the images are stitched together, the resulting mosaic may look like a tepee. In order to address 
this, a terrain-guided mosaicing process may be implemented to guide the placement of a seam 
5 line. For example, LIDAR or DEM data collected with, or analyzed from, image data may be 
processed to determine the configuration and shaping of images as they are mosaiced together. 
Thus, in some mosaiced images, a seam line may not be a straight line - instead comprising a 
seam line that shifts back and forth to snake through elevated structures. 

[0068] Referring now to Figure 8, one embodiment of an imaging process 800 is illustrated in 
10 accordance with the present invention as described above. Process 800 begins with a series 802 
of one, or more, raw collected images. Images 802 are then processed through a white-balancing 
process 804, transforming them into a series of intermediate images. Series 802 is then 
processed through anti-vignetting fimction 806 before progressing to the orthorectification 
process 808. As previously noted, orthorectification may rely on position and attitude data 810 
15 from the imaging sensor system or platform, and on DTM data 812. DTM data 812 may be 
developed from position data 810 and from, for example, USGS DTM data 814 or LIDAR data 
816. Series 802 is now orthorectified and processing continues with color balancing 818. After 
color balancing, series 802 is converted by mosaicing module 820 into compound image 822. 
Module 820 performs the mosaicing and feathering processes during this conversion. Now, one 
20 or more compound images 822 are fiirther combined in step 824, by mosaicing with a gradient 
and feathering, into image strip 826. Image strips are processed through intensity normaUzation 
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828. The now normalized strips 828 are mosaiced together in step 830, again by mosaicing with 
a gradient and feathering, rendering a finishing tiled mosaic 832. The mosaicing performed in 
step 830 may comprise a terrain-guided mosaicing, relying on DTM data 812 or LEDAR data 
816. 

5 [0069] Figure 9 illustrates diagrammatically how photos taken with the camera array assembly 
may be aligned to make an individual frame. 

[0070] Figure 10 is a block diagram of the processing logic according to certain embodiments 
of the present invention. As shown in block diagram 1000, the processing logic accepts one or 
more inputs, which may include elevation measurements 1002, attitude measurements 1004 

10 and/or photo and sensor imagery 1006. Certain inputs may be passed through an initial 
processing step prior to analysis, as is shown in block 1008, wherein the attitude measurements 
are combined with data from ground control points. Elevation measurements 1002 and attitude 
measurements 1004 may be combined to generate processed elevation data 1010. Processed 
elevation data 1010 may then be used to generate elevation DEM 1014 and DTM 1016. 

15 Similarly, attitude measurements 1006 may be combined with photo and sensor imagery 1006 to 
generate georeferenced images 1012, which then undergo image processing 1018, which may 
include color balancing and gradient filtering. 

[0071] Depending on the data set to be used (1020), either DTM 1016 or a USGS DEM 1022 
is combined with processed images 1018 to generate orthorectified imagery 1024. Orthorectified 
20 imagery 1024 then feeds into self-locking flightlines 1026. Balancing projection mosaicing 1028 
then follows, to generate final photo output 1030. 
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[0072] The present invention may employ a certain degree of lateral oversampling to improve 
output quality. Figure 11 is an illustration of a lateral oversampling pattern 1100 looking down 
from a vehicle according to certain embodiments of the present invention showing minimal 
lateral oversampling. In this illustration, the central nadir region 1102 assigned to the center 
5 camera overlaps only shghtly with the left nadir region 1 104 and right nadir region 1 106, so that 
overlap is minimized. Figure 12 is an illustration of a lateral oversampling pattern 1200 looking 
down from a vehicle according to certain embodiments of the present invention showing a 
greater degree of lateral oversampling. In this illustration, the central nadir region 1202 shows a 
high degree of overlap with left nadir region 1204 and right nadir region 1206. 

10 [0073] In addition to the use of lateral oversampling as shown in Figures 11 and 12, the 
present invention may employ flight line oversampling as well. Figure 13 is an illustration of a 
flight line oversampling pattern 1300 looking down from a vehicle according to certain 
embodiments of the present invention showing a certain degree of flight line oversampling but 
minimal lateral oversampling. Central nadir regions 1302 and 1304 are overlapped to one 

15 another along the flight line, but do not overlap laterally with left nadir regions 1306 and 1308 or 
with right nadir regions 1310 and 1312. 

[0074] Figure 14 is an illustration of flight line oversampling looking down from a vehicle 
according to certain embodiments of the present invention showing significant flight line 
oversampling as well as significant lateral oversampling. It can be seen that each of the central 
20 nadir regions 1402-1406 are significantly overlapped with one another as well as with left nadir 
regions 1408-1412 and right nadir regions 1414-1418. Left nadir regions 1408-1412 are 
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overlapped with one another, as are right nadir regions 1414-1418. Accordingly, each point on 
the surface is sampled at least twice, and in some cases as many as four times. This technique 
uses the fact that in the area of an image that is covered twice, or more, by different camera 
sensors, a doubling of the image resolution is possible in both the lateral (across path) and flight 

5 line (along path) directions for an overall quadrupHng of the resolution. In practice, the 
improvement in image/sensor resolution is somewhat less than doubled in each of the 
dimensions, approximately 40% in each dimension, or 1.4 X 1.4 = '--2 times. This is due to the 
statistical variations of the sub-pixel alignment/orientation. In effect, the pixel grid is rarely 
exactly equidistant from the overlaid pixel grid. If extremely precise lateral camera sensor 

10 alignments were made at the sub-pixel level, a quadrupling of image resolution could be reaUzed. 
[0075] Figure 15 is an illustration of a progressive magnification pattem 1500 looking down 
from a vehicle according to certain embodiments of the present invention. Central nadir region 
1502 is bounded on its left and right edges by inner left nadir region 1504 and inner right nadir 
region 1506, respectively. Inner left nadir region 1504 is bounded on its left edge by outer left 

15 nadir region 1508, while inner right nadir region 1506 is bounded on its right edge by outer right 
nadir region 1510. Note that these regions exhibit a minimal degree of overlap and oversampling 
from one to another. 

[0076] Figure 16 is an illustration of a progressive magnification pattem 1600 looking down 
from a vehicle according to certain embodiments of the present invention. Central nadir region 
20 1602 is bounded on its left and right edges by inner left nadir region 1604 and inner right nadir 
region 1606, respectively. Inner left nadir region 1604 is bounded on its left edge by outer left 
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nadir region 1608, while inner right nadir region 1606 is bounded on its right edge by outer right 
nadir region 1610. Note that, as above, these regions exhibit a minimal degree of overlap and 
oversampling from one to another. Within each of the nadir regions 1604-1610, there is a central 
image region 1614-1620 shovm shaded in grey. 

5 [0077] Figure 17 is an illustration of a progressive magnification pattem 1700 looking down 
from a vehicle according to certain embodiments of the present invention. In the center of 
pattem 1700, a left inner nadir region 1702 and a right inner nadir region 1704 overlap in the 
center. A left intermediate nadir region 1706 and a right intermediate nadir region 1708 are 
disposed partly outside of regions 1702 and 1704, respectively, each sharing an overlapping area 

10 with the respective adjacent area by approximately 50%. An outer left nadir region 1710 and an 
outer right nadir region 1712 are disposed partly outside of regions 1706 and 1708, respectively, 
each sharing an overlapping area with the respective adjacent area by approximately 50%. A 
central image region 1714 is disposed in the center of pattem 1700, comprised of the central 
portions of nadir regions 1702-1712. 

15 [0078] Figure 18 depicts a schematic of the architecture of a system 1800 according to certain 
embodiments of the present invention. System 1800 may include one or more GPS satellites 
1802 and one or more SATCOM satellites 1804. One or more GPS location systems 1806 may 
also be included, operably connected to one or more modules 1808 collecting LIDAR, GPS 
and/or X, Y, Z location data and feeding such information to one or more data capture system 

20 applications 1812. One or more data capture system appUcations 1812 may also receive spectral 
data from a camera array 1822. A DGPS 1810 may communicate with one or more SATCOM 
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satellites 1804 via a wireless communications link 1826. One or more SATCOM satellites 1804 
may, in turn, communicate with one or more data capture system applications 1812. 
[0079] One or more data capture system applications 1812 may interface with an autopilot 
1816, an SSD and/or a RealTime StitchG system 1820, which may also interact with one 
5 another. SSD 1814 may be operably connected to RealTime DEM 1818. Finally, RealTime 
DEM 1818 and RealTime StitchG 1820 may be connected to a storage device, such as disk array 
1824. 

[0080] The modules, algorithms and processes described above can be implemented in a 
number technologies and configurations. Embodiments of the present invention may comprise 
10 functional instances of software or hardware, or combinations thereof Furthermore, the modules 
and processes of the present invention may be combined together in a single functional instance 
(e.g., one software program), or may comprise operatively associated separate functional devices 
(e.g., multiple networked processor/memory blocks). All such implementations are 
comprehended by the present invention. 

15 [0081] The embodiments and examples set forth herein are presented to best explain the 
present invention and its practical application and to thereby enable those skilled in the art to 
make and utilize the invention. However, those skilled in the art will recognize that the 
foregoing description and examples have been presented for the purpose of illustration and 
example only. The description as set forth is not intended to be exhaustive or to limit the 

20 invention to the precise form disclosed. Many modifications and variations are possible in light 
of the above teaching without departing firom the spirit and scope of the following claims. 
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